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Item  13  (Concluded): 


trust  can  be  placed  in  estimates  of  the  contributions  of  experimental  tests  to  ASVAB  in  the 
prediction  of  subsequent  criteria,  even  though  the  ASVA8  scores  are  collected  some  months  prior 
to  administration  of  the  experimental  tests.  (_r  -,j  t _ 

Another  approach  to  estimating  the  contribution  of  experimental  tests  to  ASVAB  would  be  to 
readminister  ASVAB  concurrently  with  the  experimental  tests.  The  present  research  indicates  that 
this  approach  would,  on  the  average,  yield  values  comparable  to  those  which  would  be  obtained  if 
the  experimental  tests  had  been  administered  at  the  operational  testing  stations,  along  with 
ASVA8 . 

Validities  of  experimental  tests  for  concurrent  criterion  measures  administered  in  the  same 
testing  session  appear  to  be  moderately  inflated.  This  inflation  is  hypothesized  to  be  due  in 
part  to  a  few  non-cooperati ve  subjects  who  operate  at  a  reduced  level  of  effort  on  all  tests  who" 
they  are  told  that  scores  will  not  affect  their  careers.  It  is  recommended  that,  to  the  extent 
possible,  such  non-cooperative  subjects  be  identified  and  removed  from  the  validation  sample. 
Estimated  contributions  may  still  be  slightly  inflated,  but  results  from  the  present  study 
suggest  that  such  inflation  is  not  likely  to  be  large. 
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SUMMARY 


This  paper  is  concerned  with  how  to  estimate  the  contribution  of  experimental  tests  to  the  Armed  Sendees 
Vocational  Aptitude  Battery  (ASVAB)  in  accounting  for  criterion  variance.  Ordinarily,  experimental  tests  are 
administered  to  individuals  already  in  the  service  and  an  attempt  is  made  to  determine  how  much  predictive 
efficiency  these  tests  contribute  to  ASVAB  subtests,  which  were  administered  some  months  previously.  This 
approach  has  been  justly  criticized,  in  that  ability  levels  could  have  changed  between  the  time  of  ASVAB 
administration  and  experimental  test  administration.  In  the  LAMP  program,  more  serious  problems  are  faced. 
Often  an  attempt  is  made  to  determine  how  much  power  the  LAMP  tests  add  to  ASVAB  when  the  criterion 
measure  is  administered  concurrently  with  the  experimental  tests.  Thus,  important  questions  need  to  be  answered. 
How  much  do  the  abilities  of  Air  Force  applicants  change  between  the  time  of  operational  testing  and  entry  into 
service?  Would  scores  from  ability  tests  administered  at  time  of  entry  into  technical  schools  have  higher  validities 
for  course  grades  than  would  scores  collected  some  months  earlier  in  the  operational  testing  program?  How  much 
are  estimates  of  the  validity  added  by  LAMP  tests  to  ASVAB  inflated  because  of  concurrency  effects?  It  was 
possible  to  address  these  and  other  questions  in  the  present  research  because  of  the  availability  of  ASVAB 
test-retest  data  on  over  4,000  cases.  Investigation  of  ability  changes  between  test  and  retest  was  made  by 
meta- analyses  of  validities  for  technical  school  grades.  It  was  hypothesized  that  retest  scores  should  have  higher 
validities,  since  they  tap  abilities  at  tin  time  of  school  entry.  Concurrency  effects  were  analyzed  by  internal  analysis 
of  ASVAB  test-retest  data. 

With  the  exception  of  slightly  higher  retest  validities  for  ASVAB  measures  of  technical  knowledge,  the  test  and 
retest  validities  centered  at  about  the  same  level.  The  validities  of  ASVAB  retest  scores  may  have  been  slightly 
underestimated  because  of  situational  variance  during  the  experimental  retest  situation;  however,  this  situational 
variance  will  lead  to  underestimates  of  the  validity  of  any  experimental  test  for  subsequent  operational  criteria.  It 
was  concluded  that,  until  better  information  becomes  available,  the  estimated  contribution  of  experimental  tests 
to  ASVAB  in  the  prediction  of  subsequent  criteria  be  accepted  at  face  value. 

Another  approach  to  estimating  the  contribution  of  experimental  tests  to  ASVAB  would  be  to  readministered 
ASVAB  concurrently  with  the  experimental  tests.  The  present  research  indicates  that  this  approach  would,  on  the 
average,  yield  values  comparable  to  those  which  would  be  obtained  if  the  experimental  tests  had  been  administered 
at  the  operational  testing  stations  with  ASVAB. 

Validities  of  experimental  tests  for  concurrent  criterion  measures  (administered  in  the  same  testing  session)  appear 
to  be  moderately  inflated.  This  inflation  is  hypothesized  to  be  due  in  part  to  a  few  non-cooperative  subjects  who 
operate  at  a  reduced  level  of  effort  on  all  tests  when  they  are  told  that  scores  will  have  no  effect  on  their  careers. 
It  is  recommended  that,  to  the  extent  possible,  such  non-cooperative  subjects  be  identified  and  removed  from 
validation  samples.  Estimated  contributions  may  still  be  slighly  inflated,  but  results  from  the  present  research 
suggest  that  such  inflation  is  not  likely  to  be  large. 


1 


PREFACE 


Support  for  this  effort  was  provided  by  the  Air  Force  Human  Resources  Laboratory  (AFHRL),  under  contract 
F41689  S6-D-O052.  The  data  analyzed  and  reported  in  this  paper  were  originally  collected  by  Maj.  John  Welsh  and 
Capt  Randy  Massey  for  use  in  the  ASVAB  quality  control  program.  I  am  indebted  to  Mr.  Calvin  Fresne  and  Mrs. 
Janice  Herford  for  assistance  in  locating  data  files  and  reorganizing  them  for  analysis.  Ms  Carmen  Pena  provided 
assistance  in  proofing  tables  and  locating  references.  Appreciation  is  expressed  to  Dr.  William  Alley,  Dr.  Patrick 
Kyllonen,  Dr.  William  Tirre,  Dr.  Scott  Chaiken,  and  Dr.  Sidney  Irvine  for  their  suggestions  and  encouragement.  I 
also  wish  to  acknowledge  the  assistance  of  Dr.  Malcom  Ree  and  Mr.  William  Pkalen  for  comments  concerning 
analysis  alternatives.  Finally,  l  am  deeply  indebted  to  Dr.  Lloyd  Humphreys  and  Dr.  Daniel  Woltz,  both  of  whom 
read  early  drafts  of  this  paper  and  made  many  helpful  suggestions  leading  to  its  improvement. 


u 


TABLE  OF  CONTENTS 


Page 


I.  INTRODUCTION, .  1 

II.  APPROACH .  2 

III.  METHOD .  2 

IV.  RESULTS .  3 

Test-Retest  Correlations,  Means  and  Standard  Deviations .  3 

Test-Retest  Validation  Against  Technical  School  Grades .  5 

Internal  Regression  Analyses  Using  ASVAB  Test-Retest  Data . 1 0 

V.  DISCUSSION . 15 

VI.  CONCLUSIONS . . 15 

REFERENCES . 16 


LIST  OF  TABLES 


Table  Page 

1  ASVAB  Tests  and  Abbreviations .  3 

2  ASVAB  Test-Retest  Correlations . 3 

3  ASVAB  Test-Retest  Means  and  Standard  Deviations .  4 

4  Test-Retest  Correlations  with  School  Grades .  6 

5  Differences  in  Test  Validities  for  Courses,  Ordered  by  Tests  and  T-Values .  8 

6  Meta-Analyses  Results .  9 

7  Time  1  and  Time  2  Means  and  Average  Intercorrelations  for  Time  1 , 

and  Time  1-Time  2 . 10 

8  Experimental  Design . 12 

9  Squared  Multiple  Correlations  for  Model  Equations . 13 

10  Estimated  Contribution  of  'Experimental'  Tests  to  ASVAB  in  the 

Prediction  of  Selected  ‘Criterion’  Measures . 14 


iii 


Estimating  the  Contribution  of 
Experimental  Tests  to  the  Armed  Services 
Vocational  Aptitude  Battery 


I.  INTRODUCTION 

Advances  in  cognitive  theory  and  the  general 
availability  of  microcomputers  have  stimulated  re¬ 
search  during  the  last  decade  on  the  development 
of  computer-based  tests  designed  to  measure  in¬ 
dividual  differences  in  cognition.  A  question  cur¬ 
rently  being  addressed  is  whether  these  new 
theory-based  tests  will  contribute  anything  to  con¬ 
ventional  paper-and-pencil  ability  tests  in  predict¬ 
ing  subsequent  learning  and  performance  criteria. 
During  the  last  few  years,  scientists  working  in  the 
Air  Force  Learning  Abilities  Measurement  Pro¬ 
gram  (LAMP)1  have  conducted  research  involv¬ 
ing  validation  of  new  experimental  tests  against 
concurrent  or  subsequent  learning  and  perfor¬ 
mance  measures  .  A  normal  practice  is  to  obtain 
official  scores  from  the  previously  administered 
Armed  Services  Vocational  Aptitude  Battery 
(ASVAB)  and  compute  how  much  the  experi¬ 
mented  tests  contribute  to  that  battery  in  ac¬ 
counting  for  criterion  variance.  Results  from  this 
procedure  relating  to  the  incremental  validity  of 
the  experimental  tests  may  be  inflated.  The 
ASVAB  validities  may  be  attenuated  because  in¬ 
dividuals  have  changed  in  the  time  between 
ASVAB  aud  criterion  test  administrations.  Be¬ 
cause  the  new  experimental  tests  are  administered 
at  or  near  the  time  of  the  criterion  measures,  they 
are  not  subject  to  this  attenuation. 


Humphreys  and  others  (Humphreys,  1960; 
Humphreys  &  Davey,  1988;  Humphreys,  Davey, 
&  Park,  1985;  Humphreys,  Parsons,  &  Park,  1979) 
have  shown  that  when  test  scores  or  school  grades 
are  collected  on  the  same  individuals  at  several 
points  in  time,  the  intercorrelations  of  scores  or 
grades  yield  a  quasi-simplex  matrix.  That  is,  high 
correlations  are  obtained  between  successive 
scores  or  grades  across  a  short  time  period;  but 
the  longer  the  time  intervening  between  data  col¬ 
lection,  the  lower  the  correlations.  However,  it 
should  be  noted  that  these  studies  dealt  with  chil¬ 
dren  or  with  young  adults  in  college.  In  both 
instances  significant  changes  in  knowledge  across 
time  should  have  been  expected. 

It  may  be  that  the  knowledge  changes  in  Air 
Force  selectees  between  testing  and  service  entry 
are  far  less  dramatic  than  those  demonstrated  by 
Humphreys.  Over  40%  of  the  individuals  tested 
in  LAMP  enter  the  Air  Force  within  6  months  of 
operational  testing,  and  roughly  87%  come  in 
within  12  months  of  operational  testing.  During 
this  intervening  time,  many  of  these  individuals 
are  not  in  school  at  all,  although  some  are  tested 
in  high  school  and  take  additional  courses  before 
graduation  and  entry  into  the  Air  Force. 

The  only  way  to  accurately  determine  how  much 
an  experimental  test  will  contribute  to  ASVAB  in 
predicting  a  subsequent  criterion  would  be  to  ad- 


LAMP  is  a  basic  research  program  which  is  jointly  sponsored  by  the  Air  Force  Office  of  Scientific  Research  (AFOSR) 
and  the  Air  Force  Human  Resources  Laboratory  (AFHRL).  LAMP  seeks  to  understand  how  individuals  process  information  in 
order  to  perceive,  store,  remember,  solve  problems,  and  acquire  knowledge  and  skills.  An  ultimate  goal  is  to  develop  new  ability 
measures  that  can  be  used  in  the  Air  Force  Personnel  Selection  and  Classification  Program, 


1 


minister  it  along  with  the  ASVAB  in  the  opera¬ 
tional  setting.  This  is  simply  not  feasible  to  do  on 
a  routine  basis  at  the  present  time. 

A  second  approach  would  be  to  readminister  the 
ASVAB  along  with  experimental  tests  after  sub¬ 
jects  have  entered  the  Air  Force.  Then  one  could 
find  out  how  much  power  the  experimental  tests 
add  to  ASVAB  in  predicting  concurrent  or  subse¬ 
quent  criteria.  This  would  eliminate  the  time  dif¬ 
ferential  between  ASVAB  and  experimental  test 
administrations,  but  it  might  not  accurately  show 
what  the  validities  and  joint  validities  would  have 
been  had  these  measures  been  administered  in  the 
operational  testing  stations  some  months  pre¬ 
viously.  Furthermore,  insufficient  testing  time  is 
available  for  this  approach. 

The  third  approach  is  the  one  currently  being 
used  in  LAMP  research;  that  is,  to  determine  how 
much  the  scores  on  experimental  tests,  adminis¬ 
tered  at  or  near  the  time  of  criterion  data  collec¬ 
tion,  contribute  to  operational  ASVAB  scores  in 
accounting  for  variance  in  the  criterion.  This  is 
undoubtedly  the  weakest  approach,  but  the  only 
one  currently  feasible.  No  information  is  yet 
available  as  to  how  fallible  such  estimates  might 
be. 

Thus,  serious  questions  must  be  addressed  re¬ 
garding  the  LAMP  validation  process.  How  much 
do  the  abilities  of  Air  Force  applicants  change 
i-’t'vpdi  time  of  ooers'ionul  testing  and  the 
time  of  entry  into  the  service?  Would  scores  from 
ability  tests  administered  at  time  of  entry  have 
higher  validities  for  technical  school  grades  than 
would  scores  collected  some  months  earlier  in  the 
operational  testing  program?  How  much  are  es¬ 
timates  of  the  validity  contributed  by  LAMP  tpsts 
inflated  because  of  concurrency  effects?  The 
present  study  was  designed  to  provide  informa¬ 
tion  bearing  on  such  questions. 


II.  APPROACH 

During  the  1984-1987  time  period,  the  Air  Force 
readministered  the  ASVAB  to  random  samples  of 
airmen  on  the  6th  day  of  their  Basic  Military 
Training.  This  is  the  same  day  that  experimental 
LAMP  data  are  collected.  The  present  investiga¬ 
tion  consists  of  analyses  of  these  ASVAB  test-re- 
test  data  in  an  effort  to  address  the 
predictive-concurrent  validity  question. 

III.  METHOD 

ASVAB  test-retest  scores  were  available  on  all 
subtests  for  4,077  cases.  The  sample  was  divided 
into  three  subsamples:  (a)  those  who  entered  the 
Air  Force  within  6  months  after  initial  testing 
(N  =  1,774),  (b)  those  who  entered  between  6  and 
12  months  after  testing  (N  =  1,785)  and  (c)  those 
who  entered  12  or  more  months  after  testing 
(N  =  518). 

Three  types  of  analyses  were  conducted.  First, 
test-retest  correlations,  means,  and  standard  de¬ 
viations  were  computed  for  all  10  ASVAB  sub¬ 
tests.  These  data  provide  a  rough  indication  of  the 
stability  of  test  scores  across  time.  Second,  test 
and  retest  data  were  correlated  with  technical 
school  grades  for  a  variety  of  courses.  Due  to 
concurrency  effects,  one  would  expect  the  retest 
scores  to  have  higher  validities.  Finally,  each  of 
eight  ASVAB  subtests  at  Time  2  (retest)  was  se¬ 
lected  as  a  "criterion,"  and  in  each  instance,  a 
second  related  test  was  selected  as  an  "experimen¬ 
tal"  predictor.  It  was  then  possible  to  compare 
how  much  the  experimental  predictor  added  to 
"ASVAB"  (the  remaining  eight  subtests)  in  pre¬ 
dicting  the  criterion  under  two  conditions:  (a)  The 
ASVAB  was  administered  months  earlier  undej 
operational  conditions,  and  (b)  the  ASVAB  was 
readministered  concurrently  with  the  experimen¬ 
tal  and  criterion  variables.  Each  of  these  analyses 
is  discussed  below. 
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IV.  RESULTS 

Test-Retest  Correlations,  Means,  and 
Standard  Deviations 

Table  1  presents  the  names  of  the  various  sub¬ 
tests  in  ASVAB,  along  with  the  abbreviations 
which  are  used  throughout  this  paper.  Table  2 
reports  test-retest  correlations  for  the  ASVAB 
subtests,  separately  for  each  of  the  three  subsam¬ 
ples  and  the  total  sample.  This  table  also  reports 
median  internal  consistency  reliability  estimates 
for  eight  of  the  subtests,  as  reported  by  Ree,  Mul¬ 
lins,  Mathews.  &  Massey  (1982).  Table  3  reports 

Tabla  1 

ASVAS  TESTS  AND  ABBREVIATIONS 


TEST  AB8S. 

General  Science  GS 
Arithmetic  Reasoning  AR 
Word  Knout edge  UK 
Paragraph  Comprehension  PC 
Numerical  Operations  MO 
Coding  Speed  CS 
Auto- Shop  Information  AS 
Mathematics  Knowledge  MK 
Mechanical  Comprehens i on  MC 
Electrical  Information  El 


Tabic  2 


ASVAB  TEST-RETEST  CORRELATIONS 


TEST 

0-6*0$ 

6-12HOS 

>12M0$ 

TOTAL 

rll 

N 

1174 

1785 

518 

4077 

CS 

.744 

.736 

.737 

.740 

.84 

AR 

.768 

.762 

.773 

.766 

.90 

UK 

.774 

.765 

.798 

.772 

.92 

PC 

.493 

.407 

.350 

.439 

.80 

NO 

.690 

.672 

.620 

.674 

• 

CS 

.694 

.651 

.596 

.663 

* 

AS 

.800 

.808 

.776 

.800 

.88 

MK 

.839 

.819 

.817 

.828 

.87 

MC 

.716 

.704 

.714 

.710 

.86 

El 

.692 

.687 

.662 

.685 

.83 

Notes:  rll  refer*  to  internal  conaiatancy 
rat iabi l itiaa.  *  ■  not  computed. 
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lesi-retest  means  and  standard  deviations  for  the 
various  subsamples  and  total  sample. 

Data  in  Table  2  reveal  some  lack  of  stability  in 
scores  across  time,  especially  for  the  Paragraph 
Comprehension,  Coding  Speed,  Numerical  Oper 
ations,  and  Electrical  Information  subtests.  Part 
of  this  is  attributable  to  a  lack  of  internal  consis¬ 
tency  in  the  tests  themselves.  Paragraph  Compre¬ 
hension  and  Electrical  Information  have  the  two 
lowi,ai  reliability  coefficients  of  those  considered. 
Furthermore,  Numerical  Operations  and  Coding 
Speed  are  both  speeded  tests,  and  speeded  tests 
are  noted  for  their  lack  of  stability  across  time. 
The  test-retest  correlations  are  attenuated  some¬ 
what  by  the  use  of  several  test  forms.  However, 
great  pains  have  been  taken  to  produce  ASVAB 
forms  which  are  equivalent  in  terms  of  item  char¬ 
acteristics,  and  such  attenuation  is  judged  to  be  of 
minor  consequence. 

Data  for  the  three  time  periods  reported  in 
Table  2  do  not  form  a  simplex.  Although  the  cor¬ 
relations  for  the  6-12  months  group  are  slightly 
lower  that  those  for  the  0-6  months  group,  several 
of  the  correlations  for  the  over  12  months  group 
are  actually  higher  than  those  for  the  0-6  months 
group.  The  most  striking  feature  in  Table  2  is  the 
consistency  of  values  across  time.  Of  course,  these 
data  do  not  represent  correlations  on  the  same 
individuals  across  time.  They  are  from  indepen¬ 
dent  subsamples.  The  small  differences  noted 
could  be  due  entirely  to  sampling  fluctuations. 

Table  3  reports  test-retest  means  and  standard 
deviations  for  subtests,  separately  for  each  of  the 
subsamples  and  the  total  Suiii  pie.  The  test-retest 
means  show  a  remarkable  stability  across  time. 
Although  some  minor  fluctuations  can  be  ob¬ 
served  from  subtest  to  subtest,  it  appears  that,  on 
the  whole,  forgetting,  learning,  practice  effects, 
regression  effects  and  situational  variance  at  Time 
2  are  fairly  well  balanced  out.  The  standard  devi¬ 
ations  appear  to  be  generally  higher  at  Time  2,  a 
matter  which  will  be  discussed  later  in  this  paper. 

Although  the  data  in  Tables  2  and  3  suggest  little 
difference  between  test  and  retest  scores  regard¬ 
less  of  interval  times,  firm  conclusions  concerning 


TABLE  3 


ASVAB  TEST-RETEST  MEANS  AND  STANDARD  DEVIATIONS 


MEANS 


0-6 

MONTHS 

6-12 

MONTHS 

>12 

MONTHS 

TOTAL 

SAMPLE 

(N-1774) 

(N- 

1785) 

(N» 

518) 

(N-4077) 

TEST 

TIME1 

TIME2 

TIME1 

TIME2 

TIME1 

TIME2 

TIME1 

TIME2 

GS 

18.7 

18.4 

18.6 

IS. 6 

18.9 

19.1 

18.7 

18.6 

AR 

22.9 

22.1 

22.8 

21.9 

23.3 

22.6 

22.9 

22.1 

UK 

29.2 

29.4 

28.8 

29.5 

29.2 

29.9 

29.1 

29.5 

PC 

12. S 

12.4 

12.4 

12.3 

12.4 

12.5 

•>2.4 

12.4 

NO 

40.8 

40.6 

40.9 

40.3 

41.6 

41.1 

40.9 

40.5 

CS 

53.3 

54.3 

52.3 

53.4 

53.0 

53.0 

52.8 

53.7 

AS 

18.4 

18.7 

18.2 

18.5 

17.9 

18.7 

18.3 

18.6 

KK 

16.6 

16.2 

16.9 

16.3 

17.2 

16.8 

16.8 

16.3 

MC 

17.9 

18.0 

17.6 

17.8 

17.7 

18.3 

17.7 

18.0 

El 

14.1 

13.9 

13.9 

13.9 

13.8 

14.0 

14.0 

13.9 

STANDARD 

DEVIATIONS 

0-6  MONTHS 

6-12 

MONTHS 

>12 

MONTHS 

TOTAL 

SAMPLE 

(N-1774) 

(N- 

1785) 

(N- 

518) 

CN=4077> 

TEST 

TIME1 

TIME2 

TIME1 

TIME2 

TIME1 

TIME2 

TIME1 

TIME2 

GS 

3.66 

3.77 

3.53 

3 .63 

3.29 

3.56 

3.56 

3.71 

AR 

4.64 

5.35 

4.59 

5.17 

4.52 

4.93 

4.60 

5.22 

UK 

4.37 

4.29 

4.34 

4.11 

4.10 

3.90 

4.33 

4.16 

PC 

1  .93 

1.99 

1.92 

1  .92 

1.78 

1 .92 

1  .91 

1 .95 

NO 

7.24 

7.67 

7.01 

7.58 

7.01 

7.33 

7.12 

7.59 

CS 

11.94 

12.30 

11.41 

12.23 

11.60 

11.48 

11.67 

12.18 

AS 

4.39 

4.27 

4.33 

4.41 

4.41 

4.36 

4.37 

4.34 

MK 

5.04 

5.23 

4.74 

5.13 

4.68 

5.02 

4.87 

5.16 

MC 

3.98 

3.98 

4.08 

3.98 

4.00 

3.89 

4.03 

3.97 

El 

3.15 

3.37 

3.12 

3.36 

3.19 

3.44 

3.14 

3.38 
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ability  changes  across  time  cannot  be  drawn  from 
comparative  analyses  of  data  from  three  different 
subsamples.  Therefore,  emphasis  was  shifted  to 
the  second  approach,  which  involved  validation  of 
Time  1  (test)  and  Time  2  (retest)  scores  against 
technical  school  course  grades. 

Test-Retest  Validation  Against  Techni¬ 
cal  School  Grades 

The  ASVAB  subtest  test-ietesf  correlations  are 
low  enough  to  allow  for  significant  changes  to  have 
occurred  in  the  abilities  of  applicants  between  the 
time  of  operational  testing  and  the  individuals’ 
entry  into  the  Air  Force.  To  the  extent  that  such 
changes  did  occur,  one  would  expect  the  retest 
scores  to  have  higher  validities  for  technical 
school  grades.  This  follows  from  the  fact  that  the 
scores  gathered  just  prior  to  entry  into  the  techni¬ 
cal  school  should  be  the  best  indicators  ot  abilities 
at  that  time.  Two  approaches  were  taken  to  eval¬ 
uate  this  hypothesis.  One  involved  using  a  simple 
signs  test  of  the  differences  in  validity  coefficients 
yielded  by  the  test  and  retest  data.  The  second 
involved  more  sophisticated  meta-analyses. 

Signs  Tests.  ASVAB  scores  for  the  4,077  sub¬ 
jects  included  in  the  study  were  matched  with 
operational  files  to  obtain  technical  school  grades, 
fable  4  reports  test  and  retest  validities  of  the  10 
ASVAB  subtests  for  grades  in  each  of  the  20 
courses  in  which  the  matching  process  yielded  40 
or  more  subjects.  Thus,  200  pairs  of  validity  coef¬ 
ficients  were  available  to  test  the  hypothesis.  Be¬ 
cause  test  and  retest  scores  came  from  the  same 
individuals,  200  t-values  were  computed  using 
fotelling’s  formula  for  correlated  data,  one  for 
each  test-retest  validity  pair.  These  t-values  are 
reported  in  Table  5,  ordered  by  ASVAB  subtest. 
A  negative  t-value  indicates  that  a  particular  retest 
validity  coefficient  is  higher  than  its  associated  test 
validity  coefficient. 

A  cursory  inspection  of  the  data  in  Table  5  re¬ 
veals  a  weak  tendency  for  the  retest  validities  to 
be  higher.  There  are  110  negative  t-values  and  90 
positive  ones.  Nevertheless,  applying  a  simple 
signs  test  produces  a  Chi-Square  of  only  -2.225, 
which  is  non-significant  at  the  .05  level.  The  over¬ 


all  hypothesis  must  be  rejected.  It  appears  that,  in 
general,  retest  validities  are  not  significantly 
higher  than  test  validities.  Although  test-retest 
validities  did  not  differ  across  all  subtests,  it  was 
hypothesized  that  ability  changes  were  more  likely 
to  occur  among  the  ASVAB  subtests  measuring 
technical  knowledge,  than  among  those  measur¬ 
ing  general  ability  or  perceptual  speed.  Certainly 
technical  knowledge  is  subject  to  learning  and 
forgetting  across  time.  Therefore,  the  ASVAB 
subtests  were  sorted  into  three  categories  for  fur¬ 
ther  evaluation,  as  follows: 

TECHNICAL  KNOWLEDGE 
Electrical  Information 
Mechanical  Comprehension 
General  Science 
Auto-Shop  Information 

GENERAL  ABILITY 
Arithmetic  Reasoning 
Word  Knowledge 
Mathematics  Knowledge 
Paragraph  Comprehension 

PERCEPTUAL  SPEED 
Numerical  Operations 
Coding  Speed 

Data  in  Table  5  reveal  that  the  four  technical 
knowledge  subtests  yielded  52  negative  t-values 
but  only  28  positive  t-values.  In  this  instance,  the 
simple  signs  test  produces  a  Chi-Square  of  -7.212. 
which  is  significant  beyond  the  .01  level.  Thus,  the 
hypothesis  of  higher  retest  validities  for  ASVAB 
measures  of  technical  knowledge  is  supported. 
The  signs  tests  for  differences  in  the  general  ability 
and  perceptual  speed  ASVAB  measures  pro¬ 
duced  Chi-Squares  of  0.812  and  -0.366,  respec¬ 
tively,  neither  of  which  is  significant  at  the  .05 
level. 


Meta-Analyses.  As  a  second,  and  much  more 
powerful  approach,  a  series  of  mcta-analyscs  were 
conducted  for  the  three  categories  of  tests  men¬ 
tioned  above  (technical  knowledge,  general  abil¬ 
ity,  and  perceptual  speed),  and  for  all  subtests 
combined.  Results  of  these  analyses  are  presented 


5 


Table  4 


TEST-RETEST  CORRELATIONS  WITH  SCHOOL  GRADES 


81130  (APPRN .  SEOJam  SPECIALIST) 


TEST 

N 

GS 

AR 

UK 

PC 

NO 

cs 

AS 

MK 

MC 

El 

TIME1 

268 

0.507 

0.286 

0.459 

0.386 

0.110 

0  176 

0.273 

0.364 

0.182 

0.284 

T1XE2 

268 

0.482 

0.392 

0.454 

0.336 

0.170 

0.174 

0.281 

0.346 

C.130 

0.353 

43131 

(APPRN 

.  TACTICAL  AIRCRAFT  MAINTENANCE  SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

El 

TIME1 

223 

0.435 

0.416 

0.378 

0.364 

0.156 

0.112 

0.316 

o.4u: 

0.412 

0.382 

TIME2 

223 

0.394 

0.416 

0.315 

0.316 

0.106 

0.129 

0.389 

0.419 

0.358 

0.479 

81150 

(SECURITY  SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

El 

TIME1 

182 

0.448 

0.271 

0.345 

0.266 

0.223 

0.083 

0.255 

0.334 

0.326 

0.149 

TIME2 

182 

0.468 

0.336 

0.337 

0.271 

0.274 

0.169 

0.268 

0.324 

0.275 

0.257 

702308 

(APPRM.  ADMINISTRATION  SPECIALIST 

-STAFF 

SUPPORT) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

El 

TTHE1 

118 

0.354 

0.427 

0.JL6 

0.275 

-0.024 

0.150 

0.257 

0.367 

0.392 

0.332 

TIKE2 

118 

0.369 

0.433 

0.215 

0.439 

0.091 

0.142 

0.258 

0.481 

0.267 

0.251 

64530 

(APPRN 

.  INVENTORY  MANAGEMENT  SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

.  cs 

AS 

MK 

MC 

El 

TIHE1 

114 

0.264 

0.451 

0.329 

0.205 

0.037 

0.016 

0.195 

0.273 

0.059 

0.078 

TIME2 

114 

0.250 

0.345 

0.274 

0.181 

0.084 

-0.U66 

0.202 

0.323 

0.186 

0.171 

81132 

(APPRN 

.  LAW  ENFORCEMENT  SPECIALIST) 

TEST 

H 

GS 

AR 

UK 

PC 

NO 

cs 

AS 

MK 

MC 

El 

TIME1 

110 

0.447 

0.328 

0.444 

0.474 

0.126 

0.204 

0.325 

0.383 

0.372 

0.310 

TIME2 

110 

0.542 

0.406 

0.401 

0.362 

0.209 

0.167 

0.418 

0.346 

0.340 

0.455 

57130 

(APPRN 

.  FIRE 

PROTECTION  SPECIALIST) 

TEST 

h 

GS 

AR 

UK 

PC 

NO 

cs 

AS 

MK 

MC 

El 

T1ME1 

89 

0  293 

0.302 

0  253 

0.294 

0.085 

0.147 

0.221 

0.343 

0.323 

0.279 

TIHE2 

89 

0.417  0.3867 

Q.243 

0.167 

0.134 

0.063 

0.226 

0.233 

0.451 

0.333 

90230 

(APPRN 

.  MEDICAL  SERVICES  SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

El 

timei 

86 

0.482 

<1.350 

0.379 

0.310 

0.020 

-0.076 

0.399 

0.323 

0.206 

0.293 

TIME2 

86 

0.473 

0.357 

0.462 

0.286 

-0.077 

0.018 

0.508 

0.228 

0.330 

0.465 

63130 

(APPRN . 

.  FUEL 

SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

El 

TIME1 

84 

0.150 

0.432 

0.075 

0.214 

0.320 

0.Z3O 

0.221 

0.446 

0.152 

0.096 

TIME2 

84 

0.071 

0.401 

0.093 

0.240 

0.242 

0.269 

0.083 

0.414 

0.173 

0.101 

73230 

(APPRN. 

.  PERSONNEL  SPECIALIST) 

TEST 

M 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

El 

TIMEI 

74 

0.354 

0.404 

0.513 

0.334 

0.078 

-0.054 

0.142 

0.409 

0.142 

0.093 

TIHE2 

74 

0.241 

0.475 

0.441 

0.379 

0.112 

0.151 

0.123 

0.376 

0.231 

0.267 
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Table  4  (Coot.) 


TEST-RETEST  CORRELATIONS  UITH  SCHOOL  GRADES 


42632 

(APPRN 

.  JET  ENGINE  MECHANIC) 

TEST 

N 

CS 

AR 

UK 

PC 

NO 

CS 

AS 

MC 

MC 

TIME1 

73 

0.261 

0.200 

0.201 

0.327 

-0.021 

0.318 

0.329 

0.253 

0.416 

TIME2 

73 

0.348 

0.191 

0.220 

0.217 

-0.027 

0.240 

0.269 

0.256 

0.301 

43132 

(APPRN 

.  STRATEGIC  AIRCRAFT 

MAINTENANCE  SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

TIME1 

69 

0.449 

0.457 

0.402 

0.450 

0.251 

0.293 

0.264 

0.418 

0.306 

TIME2 

69 

0.569 

0.417 

0.396 

0.485 

0.348 

0.124 

0.291 

0.453 

0.371 

42335 

(APPRN 

.  AEROSPACE  GROUND  EQUIPMENT  MECHANIC) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

M( 

MC 

TIHE1 

71 

0.090 

0.446 

-0.035 

0.061 

0.375 

0.026 

0.263 

0.268 

0.232 

TIME2 

71 

0.182 

0.421 

0.055 

0.155 

0.439 

0.206 

0.255 

0.443 

0.253 

46130 

(APPRN 

.  MUNITIONS  SYSTEMS  SPECIALIST) 

TEST 

H 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

T1ME1 

55 

0.392 

0.422 

0.351 

0.343 

0.458 

0.175 

0.447 

0.366 

0.271 

TIME2 

55 

0.623 

0.398 

0.166 

0.108 

0.257 

0.094 

0.479 

0.421 

0.427 

64531 

(APPRN 

.  MATERIEL  STORAGE  AND  DISTRIBUTION 

SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

N( 

MC 

T1ME1 

52 

0.261 

0.254 

0.383 

0.349 

0.211 

0.372 

0.087 

0.161 

0.149 

TIME2 

52 

0.290 

0.146 

0.371 

0.473 

0.113 

0.196 

-0.046 

0.204 

0.110 

12230 

(APPRN 

.  AIRCREW  LIFE 

SUPPORT  SPECIALIST) 

TEST 

H 

CS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

TIME1 

52 

0.122 

0.288 

0.158 

0.152 

0.004 

0.211 

0.422 

0.255 

0.237 

TIHE2 

52 

0.278 

0.175 

0.251 

0.264 

0.090 

0.235 

0.259 

0.262 

0.281 

43133 

(APPRN 

.  AIRLIFT  AIRCRAFT  MAINTENANCE  SPECIALIST) 

TEST 

H 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

TIME1 

45 

0.402 

0.347 

0.315 

0.296 

0.183 

0.212 

0.197 

0.366 

0.349 

TIKME2 

45 

0.318 

0.321 

0.288 

0.421 

0.148 

0.236 

0.307 

0.460 

0.480 

29130 

(APPRN 

.  TELECOMMUNICATIONS 

OPERATIONS  SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

T1ME1 

43 

0.382 

0.314 

0.402 

0.144 

0.230 

0.205 

0.384 

0.320 

0.418 

T1HE2 

43 

0.519 

0.174 

0.586 

0.403 

0.288 

0.189 

0.342 

0.330 

0.432 

42330 

(APPRN 

.  AIRCRAFT  ELECTRICAL 

,  STSTEMS  SPECIALIST) 

TEST 

H 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

TIME1 

43 

0.075 

0.104 

■0.070 

0.328 

-0.262 

-0.023 

0.094 

0.233 

0.353 

TIME2 

43 

0.251 

0.212 

-0.028 

0.162 

-0.112 

0.037 

0.186 

0.170 

0.141 

60531 

(APPRN 

.  AIR  CARGO  SPECIALIST) 

TEST 

N 

GS 

AR 

UK 

PC 

NO 

CS 

AS 

MK 

MC 

TIME1 

40 

0.182 

0.347 

0.202 

0.210 

-0.043 

0.096 

-0.033 

0.269 

0.060 

TINE2 

40 

0.295 

0.334 

0.273 

0.159 

-0.040 

0.105 

0.135 

0.254 

0.136 

El 

0.307 

0.222 


Et 

0.460 

0.392 


El 

0.207 

0.330 


El 

0.386 

0.586 


El 

0.034 

0.208 


El 

0.214 

0.270 


El 

0.377 

0.320 


El 

0.419 

0.370 


El 

0.101 

0.137 


El 

0.114 

0.114 
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Tibi*  5 


DIFFERENCES  IN  TEST  VALIDITIES  FOR  COURSES,  ORDERED  IT  TESTS  AM)  T-VALUCS 


COURSE 

TST 

T 

COURSE 

TST 

T 

COURSE 

1ST 

T 

COURSE 

TST 

T 

81132 

AR 

-1.308 

81130 

CS 

0.038 

90230 

HC 

-1.732 

73230 

NO 

-0.313 

81  ISO 

AR 

-1.200 

70230 

cs 

0.090 

64530 

HC 

-1.704 

81130 

NO 

-0.190 

57130 

AR 

-1.082 

46130 

cs 

0.094 

57130 

HC 

-1.673 

60531 

NO 

0.046 

73230 

AX 

-1.046 

29130 

cs 

0.112 

46130 

HC 

-1.246 

43133 

NO 

0.236 

42330 

At 

-0.938 

81132 

cs 

0.502 

73230 

HC 

-0.973 

12230 

HO 

0.639 

70230 

At 

-0.105 

42632 

cs 

0.850 

43133 

HC 

-0.881 

64531 

HO 

0.807 

90230 

At 

-0.082 

64530 

cs 

0.896 

43132 

HC 

-0.712 

43131 

NO 

0.905 

43131 

At 

0.000 

57130 

cs 

1.002 

12230 

HC 

-0.497 

63130 

NO 

0.977 

60531 

At 

0.097 

43132 

cs 

1.591 

60531 

HC 

-0.475 

90230 

NO 

1.147 

42632 

At 

0.119 

64531 

cs 

1.823 

42335 

HC 

-0.234 

46130 

HO 

2.557 

46130 

AR 

0.223 

63130 

HC 

-0.204 

43133 

AX 

0.232 

64531 

El 

-2.054 

29130 

HC 

-0.151 

29130 

PC 

-1.702 

42335 

AX 

0.320 

46130 

El 

-2.008 

64531 

HC 

0.351 

70230 

PC 

-1.701 

63130 

At 

0.485 

90230 

El 

-1.972 

81132 

HC 

0.378 

81130 

PC 

-1.190 

43132 

At 

0.485 

43131 

El 

-1.946 

81150 

HC 

0.921 

64531 

PC 

-1.091 

81130 

At 

1.051 

73230 

E! 

-1.849 

43131 

HC 

0.973 

42335 

PC 

-0.801 

64531 

AR 

1.110 

81150 

El 

-1.824 

81130 

HC 

1.086 

43133 

PC 

-0.768 

12230 

AR 

1.184 

81132 

E! 

-1.675 

42632 

HC 

1.361 

12230 

PC 

-0.752 

29130 

AR 

1.371 

81130 

El 

-1.497 

42330 

HC 

1.489 

73230 

PC 

-0.396 

64530 

AR 

1.874 

42335 

El 

-1.305 

70230 

HC 

1.722 

43132 

PC 

-0.287 

64530 

Et 

-1.218 

63130 

PC 

-0.231 

81132 

AS 

-1.887 

63130 

El 

-1.097 

42335 

HK 

-2.421 

81150 

PC 

•0.068 

90230 

AS 

-1.8Z3 

57130 

El 

-0.628 

70230 

HK 

-1.928 

90230 

PC 

0.194 

60531 

AS 

-1.787 

12230 

El 

-0.531 

43133 

HK 

•0.898 

64530 

PC 

0.217 

70230 

AS 

-1 .644 

4Z330 

El 

-0.271 

64530 

HK 

-0.865 

60531 

PC 

0.321 

43131 

AS 

-1.579 

60531 

El 

0.000 

46130 

HK 

-0.621 

43131 

PC 

0,740 

43133 

AS 

-0.942 

43133 

El 

0.384 

64531 

HK 

-0.558 

42632 

PC 

1.009 

43132 

AS 

-0.376 

29130 

El 

0.417 

43131 

HK 

-0.545 

4Z330 

PC 

1.143 

46130 

AS 

-0.356 

43132 

El 

0.795 

43132 

HK 

-0.500 

57130 

PC 

1.158 

81150 

AS 

-0.260 

42632 

El 

0.810 

29130 

HK 

-0.135 

81132 

PC 

1.304 

64530 

AS 

-0.121 

70230 

El 

1.131 

12230 

MC 

-0.082 

46130 

PC 

1.609 

81130 

AS 

-0.094 

42632 

HK 

-0.037 

57130 

AS 

-0.075 

81130 

GS 

-2.717 

60531 

HK 

0.120 

29130 

VIC 

-2.436 

42330 

AS 

0.094 

46130 

GS 

-2.619 

81150 

MC 

0.271 

12230 

UK 

-1.128 

42335 

AS 

0.095 

12230 

CS 

-1.699 

63130 

HK 

0.447 

90230 

UK 

-1.125 

73230 

AS 

0.279 

81132 

GS 

-1.630 

81130 

HK 

0.511 

42335 

UK 

-0.869 

29130 

AS 

0.415 

57130 

CS 

•1.628 

73230 

HK 

0.541 

60531 

UK 

-0.835 

42632 

AS 

0.769 

43132 

cs 

-1.500 

42330 

HK 

0.652 

42632 

UK 

-0.238 

63130 

AS 

1.491 

29130 

GS 

-1.390 

81132 

HK 

0.762 

42330 

UK 

-0.213 

64531 

AS 

1.582 

42330 

GS 

-1.245 

90230 

HK 

1.486 

43132 

UK 

0.082 

12230 

AS 

2.091 

60531 

CS 

-1.183 

57130 

MK 

2.059 

63130 

UK 

0.094 

42335 

GS 

-0.994 

64531 

UK 

0.112 

42335 

cs 

-1.927 

42632 

GS 

-0.831 

81132 

NO 

-1.305 

57130 

UK 

0.125 

73230 

CS 

-1.728 

81150 

GS 

•0.416 

70230 

NO 

-1.235 

81150 

UK 

0.161 

61150 

CS 

-1.231 

70230 

GS 

-0.347 

42330 

NO 

-1.147 

43133 

UK 

0.251 

90230 

cs 

-1.028 

64531 

GS 

-0.294 

43132 

NO 

-0.970 

81130 

UK 

0.547 

42330 

cs 

-0.470 

90230 

GS 

0.130 

81150 

NO 

-0.894 

81132 

UK 

0.863 

63130 

cs 

-0.442 

64530 

GS 

0.230 

42335 

NO 

-0.638 

73230 

UK 

0.955 

43131 

cs 

-0.291 

43133 

GS 

0.663 

64530 

NO 

•0.605 

64530 

UK 

0.993 

12230 

cs 

-0.220 

43131 

GS 

0.918 

57130 

NO 

-0.598 

43131 

UK 

1.519 

43133 

cs 

-0.172 

63130 

GS 

0.977 

42632 

NO 

-0.550 

46130 

UK 

1.608 

60531 

cs 

-0.069 

73230 

cs 

1.200 

29130 

NO 

-0.546 

70230 

UK 

1.884 

8 


T abl «  6 


META- ANALYSES  RESULTS 


20  SCHOOLS 


TEST  GROUP 

T  1 

T2 

R12 

OF 

T 

Tech.  Knoul«da« 

0.286 

0.322 

0.737 

7321 

-4.499 

C«n«r« l  Abl l f  ty 

0.336 

0.335 

0.727 

7321 

0.124 

Perceptual  Speed 

0.141 

0.148 

0.668 

3659 

-0.526 

Total 

0.280 

0.294 

0.720 

18307 

-2.662 

69  SCHOOLS 


TEST  GROUP 

T  1 

T2 

R12 

OF 

T 

Tech .  Knowledge 

0.295 

0.321 

0.737 

11337 

-4-047 

General  Ability 

0.315 

0.315 

0.727 

11337 

0.000 

Perceptual  Speed 

0.124 

0.136 

0.668 

5667 

-1-120 

Total 

0.270 

0.283 

0.720 

28347 

-3.065 

Note:  Values  under  Tt  and  T2  are  average  validity  coefficients 
computed  using  Fisher's  z- transformation  and  weighted  by 
N's.  The  R12  column  is  the  average  time  1-time  2  test 
intercorrelat loos,  again  computed  using  Fisher's  z's 
and  weighted  by  N's.  The  last  column  reports  Hotelling's 
t's  based  on  correlated  data  for  the  dlfferencss  in  test 
and  retest  average  validities  within  each  category. 


in  Table  6,  not  only  for  data  in  the  20  schools 
reported  in  Table  5,  but  for  all  schools  in  which 
the  matching  process  yielded  15  or  more  subjects. 
The  general  approach  was  to  treat  data  from  each 
school  as  being  from  a  separate  study.  The  data 
were  combined  using  a  method  recommended  by 
Hedges  and  Olkin  (1985,  pp.  230-232).  First,  the 
test  and  retest  validities  within  each  category  were 
converted  to  Fisher’s  z’s  and  weighted  by  N-3. 
Next,  the  weighted  z’s  were  averaged,  and  the 
averages  were  converted  back  into  correlations. 
Finally,  Hotelling’s  t’s  were  computed  for  the  dif¬ 
ferences  in  test  and  retest  average  correlations 
within  each  subtest  category. 

The  data  in  Table  6  reveal  that  there  are  no 
significant  differences  between  Time  1  (test)  and 
Time  2  (retest)  validities  for  the  ASVAB  subtests 
associated  with  either  general  learning  ability  or 


perceptual  speed.  Subtest  validities  for  the  tech¬ 
nical  knowledge  area  are  significantly  higher  at 
Time  2,  both  in  the  20-school  and  69-school  sam¬ 
ples,  but  the  magnitudes  of  the  differences  are 
very  small.  The  average  retest  validities  across  all 
schools  in  both  the  20-school  and  69-school  sam¬ 
ples  are  significantly  higher  at  Time  2,  but  this  was 
in  the  presence  of  many  degrees  of  freedom,  and 
due  solely  to  differences  in  the  technical  knowl¬ 
edge  subtests.2 

In  summary,  analyses  of  the  zero-order  validities 
present  a  clear  indication  that  ability  changes  for 
Air  Force  applicants  between  time  of  testing  and 
entry  into  the  service  are  restricted  to  measures  of 
technical  knowledge,  and  are  relatively  minor. 


Some  observers  may  object  to  the  meta-analysis  approach  because  subjects  attending  the  various  schools  cannot  be 
assumed  to  have  been  drawn  from  the  same  population.  This  is  certainly  true,  since  the  entry  ability  level  requirements  vary 
from  course  to  course.  In  order  to  respond  to  this  possible  criticism,  all  validity  values  in  the  20-school  sample  were  corrected 
for  restriction  in  range  due  to  selection,  and  the  average  validities  were  recomputed.  The  revised  Time  1-Time2  average 
validities  were:  technical  knowledge,  .405  vs.  .425.;  general  ability.  501  vs.  .485;  perceptual  speed,  .197  vs  .215,  and  total,  .408  vs. 
411  These  differences  appear  to  be  less  than  those  computed  for  the  uncorrected  validities.  Hotelling  t's  were  not  computed 
because  of  problems  in  dealing  with  double  curtailments  associated  with  the  intercorrelation  terms. 
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Regression  Analyses.  Another  way  of  evaluating 
the  impact  of  ability  changes  is  to  determine  the 
differences  in  these  multiple  correlations  against 
technical  school  grades  which  are  obtained  using 
Time  1  and  Time  2  data.  This  was  accomplished 
for  each  of  the  20  technical  schools  courses  in 
which  the  matching  process  yielded  40  or  more 
subjects.  Again,  the  mean  multiple  R’s  were  based 
upon  weighted  z’s.  The  results  yielded  a  multiple 
R  of  546  for  Time  1  (test)  and  539  for  Time  2 
(retest).  Thus,  even  considering  the  joint  action 
of  all  ASVAB  subtests,  there  is  no  indication  of 
appreciable  loss  of  predictive  power  across  time. 

Internal  Regression  Analyses  Using 
ASVAB  Test-Retest  Data 

It  appears  from  the  preceding  analyses  that,  if 
ability  changes  occur  for  Air  Force  selectees  be¬ 
tween  time  of  operational  testing  and  time  of  entry 
into  the  service,  the  changes  are  associated  with 
technical  knowledge  and  are  relatively  minor  Es¬ 
timates  of  the  value  of  new  tests  in  adding  power 
to  the  operational  ASVAB  in  predicting  technical 
school  grades  would  appear  to  be  fairly  trustwor¬ 
thy.  However,  on  many  occasions  LAMP  experi¬ 


mental  tests  are  validated  against  intermediate 
learning  criteria  which  are  administered  in  the 
same  experimental  testing  session.  It  is  entirely 
possible  that  the  correlations  between  the  experi¬ 
mental  tests  and  criterion  measures  are  inflated 
because  of  test  covariance  associated  with  motiva¬ 
tional  decrements  present  in  an  experimental  test¬ 
ing  situation.  It  has  been  conjectured  that  some 
individuals  operate  with  decreased  motivation 
when  they  are  told  that  the  test  results  will  be  used 
only  for  purposes  of  research,  and  this  negatively 
impacts  their  performance  on  both  experimental 
and  criterion  measures  administered  during  an 
experimental  testing  session.  This  type  of  behavior 
would  be  expected  to  increase  the  correlations 
between  the  experimental  and  criterion  measures. 
In  the  case  of  LAMP  validation  studies,  where  the 
contribution  of  experimental  tests  to  ASVAB  is 
being  evaluated  against  a  concurrent  criterion 
measure,  it  is  important  to  know  how  much  of  the 
computed  contribution  might  be  associated  with 
this  type  of  situational  variance. 

Data  indicating  existence  of  the  problem  are 
shown  in  Table  7.  The  last  three  columns  of  this 
table  report  the  average  correlation  (computed 
using  Fisher’s  z-transformations)  of  each  ASVAB 


TABLE  7 

TIME1.TIME2  MEANS  AND  AVERAGE  INTERCORRELATIONS  FOR  TIME1.TIME2  AND  TIME1-TIME2 

TOTAL  SAMPLE  <N»4077> 

MEAN  SCORES  STD. DEVIATIONS  AVG. INTERCORRELATIONS 


TEST 

TIME1 

TIME2 

TIME1 

TIME2 

T1-T1 

T1-T2 

T2-T2 

GS 

18.66 

18.59 

3.56 

3.71 

0.375 

0.377 

0.393 

AR 

22.89 

22.08 

4.60 

5.22 

0.364 

0.377 

0.401 

UK 

29.05 

29.50 

4.33 

4.16 

0.322 

0.324 

0.357 

PC 

12.44 

12.37 

1.91 

1.95 

0.285 

0.278 

0.314 

NO 

40.91 

40.52 

7.12 

7.59 

0.131 

0.144 

0.193 

CS 

52.83 

53.70 

11.67 

12.18 

0.133 

0.130 

0.159 

AS 

18.25 

18.62 

4.37 

4.34 

0.241 

0.232 

0.267 

MK 

16.81 

16.31 

4.87 

5.16 

0.368 

0.365 

0.392 

MC 

17.74 

17.96 

4.03 

3.97 

0.349 

0.343 

0.350 

El 

14.00 

13.94 

3.14 

3.38 

0.320 

0.300 

0.359 

AVG. 

24.36 

24.36 

4.96 

5.17 

0.291 

0.290 

0.320 
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subtest  with  the  other  nine  subtests,  separately  foi 
Time  1,  for  Time  2,  and  between  Time  1  and 
Time  2.  Notice  that  the  Time  1-Time  2  correla¬ 
tions  are  almost  equivalent  to  the  Time  1-Time  1 
correlations,  providing  additional  evidence  on  the 
lack  of  ability  changes  across  time.  However,  no¬ 
tice  that  the  Time  2-Time  2  correlations  are  sys¬ 
tematically  higher  than  either  the  Time  1-Time  1 
or  the  Time  1-Time  2  correlations.  This  may  be 
attributable  to  common  situational  variance  asso¬ 
ciated  with  the  experimental  testing  situation  at 
Time  2.  That  is,  some  individuals  were  either 
tired,  under  stress,  or  unmotivated  during  the 
Time  2  experimental  retesting  session,  and  they 
systematically  made  low  scores  on  all  subtests. 
One  would  expect  this  factor  to  result  in  lower 
means,  but  comparing  the  means  from  Time  1  to 
those  from  Time  2  is  meaningless  because  of  the 
interacting  factors  of  learning,  forgetting,  prac¬ 
tice,  and  regression  effects  mentioned  previously. 

The  standard  deviations  appear  to  be  generally 
higher  at  Time  2,  and  this  too  is  attributed  to 
situational  variance  present  in  the  experimental 
testing  session.  Poorly  motivated  subjects  could 
actually  score  so  low  that  they  would  not  have  been 
accepted  into  the  Air  Force  had  they  performed 
at  the  same  level  during  the  operational  adminis¬ 
tration. 

The  fact  that  situational  variance  in  an  experi¬ 
mental  testing  session  increases  test  inter- 
correlations  has  implications  for  the  LAMP 
program.  It  indicates  that  the  correlations  ob¬ 
served  between  experimental  tests  and  criterion 
tests  administered  in  the  same  experimental  test¬ 
ing  session  are  likely  to  be  inflated.  Furthermore, 
the  computed  contribution  of  the  experimental 
tests  to  ASVAB  scores  collected  some  months 
previously  would  also  be  inflated.  The  question  is 
how  seriously  such  estimates  are  inflated. 

The  ASVAB  test-retest  data  were  used  to 
roughly  evaluate  the  amount  of  inflation  likely  in 
estimating  the  contribution  of  experimental  tests 
to  ASVAB.  The  approach  was  to  use  each  of  eight 
ASVAB  subtest  scores  at  Time  2  as  a  criterion 
measure.  In  each  instance,  a  related  subtest  was 


selected  as  being  an  experimental  predictor  vari¬ 
able,  while  the  remaining  eight  ASVAB  subtest 
scores  at  Time  1  and  again  at  Time  2  were  treated 
as  though  they  were  the  complete  ASVAB.  Three 
comparisons  were  evaluated  for  each  of  the  eight 
criterion  measures  as  defined  by  the  experimental 
design  shown  in  Table  8. 

Table  9  reports  the  R-squares  for  each  of  the  six 
comparison  equations,  separately  for  eight  cri¬ 
teria.  The  NO  and  CS  subtests  were  not  used  as 
criteria  in  this  analysis  because  of  their  unreliabil¬ 
ity,  and  because  they  tend  to  correlate  only  with 
each  other. 

Table  10  summarizes  differences  between  Com¬ 
parisons  1, 2,  and  3  from  Table  9.  Comparison  1  is 
the  estimated  contribution  of  an  experimental  test 
to  ASVAB,  when  ASVAB  is  administered  at  Time 
1,  and  data  for  the  experimental  test  and  criterion 
data  are  collected  at  Time  2  in  the  same  experi¬ 
mental  testing  session.  Comparison  2  reports  the 
actual  contribution  of  the  experimental  test  to 
ASVAB.  In  this  latter  instance,  the  experimental 
test  and  ASVAB  data  are  collected  at  Time  1,  and 
the  criterion  data  are  collected  at  Time  2.  The 
difference  between  Comparison  1  and  Compari¬ 
son  2  estimates  reflects  the  degree  of  inflation  in 
the  estimated  contribution  of  an  experimental  test 
to  ASVAB,  when  both  the  experimental  test  and 
criterion  data  are  collected  concurrently  in  the 
same  experimental  testing  session.  It  appears  that 
such  inflation  is  not  very  serious.  As  shown  in 
Table  10,  the  average  estimated  variance  contri¬ 
bution  across  all  eight  criteria  in  the  total  sample 
is  12.8%,  while  the  actual  contribution  is  10.4%. 
Application  of  an  18.75%  discount  would  bring 
the  average  estimated  contribution  into  align¬ 
ment. 

The  data  on  Comparison  3  presented  in  Table  10 
suggest  that  if  ASVAB  were  readministered  along 
with  the  experimental  tests  and  criterion  mea¬ 
sures,  the  estimated  contribution  of  the  experi¬ 
mental  tests  to  ASVAB  would  generally  not  be 
inflated.  If  sufficient  testing  time  were  available, 
this  might  be  the  preferred  approach. 
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Table  8 


EXPERIMENTAL  DESIGN 

T*  CRITERION  (ONE  OF  EIGHT  ASVA8  TESTS). 

X-  PREDICTOR  TEST  BEING  EVALUATED  (A  RELATED  ASVAB  TEST). 

ASVAB-  REMAINING  8  TESTS  IH  ASVAB 

TIME1-  TIME  OF  OPERATIONAL  ADMINISTRATION  OF  ASVAB 

TIME 2*  6TH  OAT  OF  BASIC  TRAINING.  SAME  DAT  AS  LAMP  TESTS  ARE  ADMINISTERED. 


COMPARISON  1 

R- SQUARED 1  Y(T1ME2)  AS  CRITERION 

XCTIME2)  AW)  ASVAB(TIMEI)  AS  PREDICTORS 

R-SQUARED2  Y(TIME2)  AS  CRITERION 

ASVA8(TIME1)  AS  PREDICTORS 

(l-SQUAPEDt)  •  (R-SOUARED2)'  ESTIMATED  UNIQUE  CONTRIBUTION  OF  EXPERIMENTAL  T>=ST  TO  ASVAB  IN  ‘ 
ACCOUNTING  FOR  CRITERION  VARIANCE.  THIS  IS  THE  UAY  THE  AIR  FORCE  AND  OTHER  SERVICES  ESTIMATE  THE 
VALUE  OF  NEU  TESTS.  THIS  ESTIMATE  IS  KNOWN  TO  BE  BIASED  IH  AH  UPWARD  DIRECTION,  BUT  HOU  MUCH  IS 
UNKNOWN. 


COMPARISON  2 

R  -  SQUARED}  Y(TIME2)  AS  CRITERION 

X(TIMEt)  AND  ASVAS(TIMEI)  AS  PREDICTORS 

R- SQUARED  A  YCTIHE2)  AS  CRITERION 

ASVAB(TINEI)  AS  PREDICTORS 

(R-SQUARED3)  -  (R-SQUAREDA)*  ACTUAL  UNIQUE  VARIANCE  CONTRIBUTED  BY  EXPERIMENTAL  TEST  TO  ASVAB  IN 
ACCOUNTING  FOR  CRITERION  VARIANCE,  WHEN  THE  EXPERIMENTAL  TEST  IS  ADMINISTERED  ON  THE  SAME  OCCASION 
AS  THE  OPERATIONAL  ASVAB. 


COMPARISON  3 

R- SQUARED 5  YCTIME2)  AS  CRITERION 

X(TIME2)  AND  ASVAB(TIME2)  AS  PREDICTORS 

R-SQUARED8  Y(TIME2)  AS  CRITERION 

ASVAB(TIME2)  AS  PREDICTORS 

(R-SQUARED5)  •  (R-SQUARED6)*  UNIQUE  VARIANCE  CONTRIBUTED  BT  THE  EXPERIMENTAL  TEST  TO  ASVAB  IN 
ACCOUNTING  FOR  THE  CRITERION  VARIANCE,  WHEN  ASVAB  IS  REAOMI HI  STEREO  CONCURRENTLY  WITH  THE  CRITERION 
AND  EXPERIMENTAL  TESTS. 
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Table  9 


SQUARED  MULTIPLE  CORRELATIONS  FOR  COMPARISON  EQUATIONS 


TOTAL  SAMPLE  (N«4077)  0-6MOS  SAMPLE  (N*1774> 


TESTS 

EQU1 

EQU2&4 

EQU3 

EQU5 

EQU6 

EQU1 

EQU2&4 

EQU3 

EQU5 

EQU6 

R-SO 

R-SQ 

R-SQ 

R-SQ 

R-SQ 

R-SQ 

R-SQ 

R-SO 

R-SQ 

R-SQ 

C  “  r 

GS-UK 

0.511 

0.384 

0.503 

0.536 

0.442 

0.511 

0.398 

0.510 

0.531 

0.448 

AR-MK 

0.527 

0.342 

0.487 

0.550 

0.390 

0.549 

0.362 

0.498 

0.573 

0.416 

UK-GS 

0.464 

0.326 

0.431 

0.486 

0.383 

0.477 

0.357 

0.456 

0.504 

0.417 

PC-UK 

0.318 

0.199 

0.275 

0.342 

0.260 

0.355 

0.237 

0.310 

0.376 

0.283 

AS-MC 

0.468 

0.382 

0.445 

0.486 

0.418 

0.463 

0.373 

0.436 

0.482 

0.417 

KZ-AR 

0.556 

0.383 

0.548 

0.550 

0.398 

0.573 

n  -mf 

0.537 

0.569 

0.415 

NC-AS 

0.453 

0.364 

0.426 

0.471 

0.401 

0.448 

0.357 

0.426 

0.471 

0.404 

El -AS 

0.528 

0.422 

0.517 

0.531 

0.431 

0.518 

0.412 

0.516 

0.526 

0.430 

6- 

12KOS  SAMPLE  <N= 

1785) 

OVER  12MOS 

SAMPLE 

04*518) 

TESTS 

EOU1 

EQU214 

EQU3 

EQU5 

EQU6 

EOU1 

EQU2&4 

EQU3 

EQUS 

EQU6 

R-SQ 

R-SQ 

R-SQ 

R-SQ 

R-SQ 

R-SQ 

R-SQ 

R-SQ 

R-SO 

R-SQ 

C  -  P 

GS-UK 

0.513 

0.376 

0.506 

0.535 

0.430 

0.510 

0.381 

0.496 

0.564 

0.467 

AR-MK 

0.516 

0.340 

0.481 

0.540 

0.379 

0.479 

0.276 

0.480 

0.503 

0.329 

UK-CS 

0.451 

0.301 

0.410 

0.474 

0.356 

0.468 

0.326 

0.436 

0.465 

0.357 

PC-VIIC 

0.279 

0.158 

0.245 

0.301 

0.224 

0.324 

0.221 

0.277 

0.384 

0.329 

AS-MC 

0.478 

0.394 

0.457 

0.492 

0.421 

0.463 

0.386 

0.438 

0.495 

0.434 

MK-AR 

0.551 

0.385 

0  537 

0.542 

0.398 

0.525 

0.322 

0.548 

0.506 

0.331 

MC-AS 

0.452 

0.364 

0.423 

0.474 

0.401 

0.472 

0.392 

0.442 

0.474 

0.416 

El -AS 

0.528 

0.426 

0.509 

0.526 

0.417 

0.573 

0.470 

0.564 

0.578 

0.486 

MOTES:  Equations  are  numbered  aa  they  are  described  in  the  experimental  design. 
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Table  10 


ESTIMATED  CONTRIBUTION  Of  EXPERIMENTAL  TESTS  TO  ASVAft  IN  THE  PREDICTION 
OF  SELECTED  CRITERION  MEASURES 


TOTAL  SAMPLE  <N«4,077)  0-6NCS  SAMPLE  <N*>1,774> 


COMP,  1 

COMP.  2 

COMP.  3 

COMP.  1 

COMP.  2 

COMP. 3 

CRI 

PRE 

EQ1-EQ2 

E03-E02 

EQS-E04 

EQ1-EQ2 

E03-EQ2 

EQS-EQ4 

GS 

UK 

12.7 

11.9 

9.4 

11.3 

11.2 

8.3 

AR 

MK 

18.5 

U.5 

16.0 

18.7 

13.6 

15.7 

UK 

GS 

13.8 

10.5 

10.3 

12.0 

9.9 

S.r 

PC 

UK 

11.9 

7.6 

8.2 

11.8 

7.3 

9.3 

AS 

MC 

8.6 

6.3 

6.8 

9.0 

6.3 

6.5 

MK 

AR 

17.3 

16.5 

15.2 

17.9 

16.5 

15.4 

MC 

AS 

8.9 

6.2 

7.0 

9.1 

6.9 

6.7 

El 

AS 

10.6 

9.5 

10.0 

10.6 

10.4 

9.6 

MEAN 

12.8 

10.4 

10.4 

12.6 

10.3 

10.0 

6-12MGS 

SAMPLE 

(N«1 , 785) 

OVER 

12MOS  SAMPLE 

CN-518) 

COMP.  1 

COMP.  2 

COMP.  3 

COMP.  1 

COMP.  2 

COMP.  3 

CRI 

PRE 

EQ1-EQ2 

EQ3-E02 

EQ5-EQ4 

bOI -EQ2 

EQ3-EQ2 

EQ5-E04 

GS 

UK 

13.7 

13.0 

10.5 

12.9 

11.5 

9.7 

AR 

MK 

17.6 

14.1 

16.1 

20.3 

20.4 

17.4 

UK 

GS 

15.0 

10.9 

11.8 

14.2 

11.0 

10.8 

PC 

UK 

12.1 

8.7 

7.7 

10.3 

5.0 

5.5 

AS 

MC 

8.4 

6.3 

7.1 

7.7 

5.2 

6.1 

MK 

AR 

16.6 

15.2 

14.4 

20.3 

22.6 

17.5 

MC 

AS 

8.8 

5.9 

7.3 

8.0 

5.0 

5.8 

El 

AS 

10.2 

8.3 

10.9 

10.3 

9.4 

9.2 

MEAN 

12.8 

10.3 

10.7 

13.0 

11.3 

10.3 

NOTES:  Reported  values  are  differences  in  R-squares.  Equations  are  numbered 
as  they  are  described  in  the  experimental  design. 

COMP.  1-  CRITERION  t  EXPERIMENTAL  TEST  AT  TIME  2,  ASVAB  AT  TIME  1 

Criterion  t  experimental  test  at  T (me  2,  ASVAB  at  Time  1. 
COMP.  2-  CRITERION  AT  TIME  2,  EXPERIMENTAL  TEST  AND  ASVAB  AT  TIME  1 

Criterion  at  Time  2,  experimental  test  and  ASVAB  at  Time  1. 
COfP.  3-  CRITERION,  EXPERIMENTAL  TEST,  l  ASVAB  AT  TIME2 

Criterion,  experimental  test  8  ASVAB  at  Time  2. 
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V.  DISCUSSION 


The  basic  problem  addressed  in  this  paper  is 
how  to  estimate  the  contribution  of  an  experimen¬ 
tal  test  to  ASVAB  in  predicting  a  criterion  mea¬ 
sure.  The  only  sure  method  would  be  to 
administer  the  experimental  test  along  with  the 
ASVAB  at  official  testing  stations  under  opera¬ 
tional  conditions.  Then  one  could  estimate  the 
contribution  of  the  experimental  test  by  evaluating 
the  difference  in  R-squares  yielded  by  a  full  model 
(containing  the  ASVAB  subtests  and  the  experi¬ 
mental  variable  as  predictors)  and  a  restricted 
model  (using  only  ASVAB  subtests  as  predic¬ 
tors).  Because  a  large  amount  of  additional  test¬ 
ing  time  in  the  operational  environment  would  be 
required  by  this  approach,  it  is  usually  not  feasible. 

In  the  typical  situation,  experimental  tests  are 
administered  to  individuals  already  in  the  service 
and  an  attempt  is  made  to  see  how  much  predic¬ 
tive  efficiency  is  added  by  these  experimental  tests 
to  ASVAB  subtests  administered  some  months 
previously.  This  approach  has  been  criticized,  in 
that  ability  levels  could  have  changed  between  the 
time  of  ASVAB  testing  and  experimental  test  ad¬ 
ministration.  The  present  study  is  an  attempt  to 
evaluate  the  changes  in  the  abilities  of  Air  Force 
personnel  between  the  time  of  their  selection  and 
their  entry  into  service. 

With  the  exception  of  slightly  higher  retest  valid¬ 
ities  for  ASVAB  measures  of  technical  knowl¬ 
edge,  the  test  and  retest  validities  centered  at 
about  the  same  level.  Although  the  validities  of 
Time  2  ASVAB  scores  may  have  been  slightly 
underestimated  due  to  variance  associated  with  a 
reduced  level  of  motivation  for  some  examinees 
during  the  experimental  testing  session,  this  same 
factor  will  lead  to  underestimates  of  the  validity  of 
any  experimental  test  for  subsequent  operational 
criteria.  It  is  concluded  that  reasonable  trust  can 


be  placed  in  estimates  of  the  contributions  of 
experimental  tests  to  ASVAB  in  the  prediction  of 
school  grades,  even  though  the  ASVAB  scores  are 
collected  some  months  prior  to  administration  of 
the  experimental  tests. 

Results  of  this  study  also  address  a  potentially 
more  serious  problem.  Experimental  test  scores 
and  intermediate  learning  criterion  scores  are 
often  collected  in  the  same  experimental  testing 
session.  In  this  case,  the  correlation  computed 
between  experimental  and  criterion  test  scores 
can  be  inflated  by  variance  associated  with  the 
experimental  testing  situation,  and  the  estimated 
contributions  of  the  experimental  tests  to  ASVAB 
are  likely  to  be  inflated.  Results  from  analyses 
conducted  using  the  ASVAB  test-retest  data  indi¬ 
cate  that  such  inflation  does  indeed  exist,  but  it  is 
not  large.  The  average  estimated  contribution  of 
an  experimental  test  to  ASVAB  across  eight  cri¬ 
terion  measures  was  12.8%,  while  the  actual  com¬ 
puted  contribution  when  the  experimental  tests 
were  administered  along  with  ASVAB  at  Time  1 
was  10.4%. 

Another  approach  to  estimating  the  contribu¬ 
tion  of  an  experimental  test  to  ASVAB  would  be 
to  readminister  ASVAB  concurrently  with  the 
experimental  test.  Data  in  this  study  indicate  that 
this  approach  would,  on  the  average,  yield  values 
comparable  to  those  which  would  be  obtained  if 
the  experimental  test  had  been  administered  at 
the  testing  stations,  along  with  ASVAB. 

VI.  CONCLUSIONS 


With  regard  to  validation  studies  against  concur¬ 
rent  criteria,  some  inflation  in  the  estimated  con¬ 
tribution  of  experimental  tests  to  ASVAB  should 
be  expected.  This  appears  to  be  due  to  a  subset  of 
individuals  who,  because  of  factors  occurring  in  an 
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experimental  testing  situation,  do  poorly  on  both 
experimental  tests  and  concurrently  administered 
criterion  measures.  Even  if  no  adjustments  are 
made,  the  degree  of  inflation  in  the  estimated 
contributions  is  not  large.  Results  from  the  pres¬ 
ent  investigation  suggest  that  an  18.8%  reduction 
in  the  estimated  contribution  would  bring  such 
estimates  closer  to  a  ;tual  contributions.  In  many 
studies,  such  discounts  may  not  be  necessary,  if  an 
attempt  is  made  to  identify  and  remove  unmoti¬ 
vated  subjects  from  analyses.3 


Until  better  information  becomes  available,  it  is 
recommended  that  the  estimated  contribution  of 
experimental  tests  to  ASVAB  in  the  prediction  of 
subsequent  criteria  be  accepted  at  face  value.  In 
the  case  of  concurrent  criteria,  it  is  recommended 
that  an  attempt  be  made  to  eliminate  obviously 
unmotivated  subjects  from  analyses  (such  as  those 
who  score  at  or  near  chance  levels  on  all  vari¬ 
ables).  Estimated  contributions  may  still  be 
slightly  inflated,  but  present  results  suggest  that 
such  inflation  is  not  likely  to  be  large. 
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There  is  a  special  problem  in  evaluating  tests  of  information  processing  speed.  No  comparable  tests  were  evaluated  in  the 
present  study,  and  there  is  reason  to  believe  that  highly  speeded  tests  are  the  ones  most  likely  to  suffer  from  situational  variance 
during  experimental  testing  sessions.  It  is  especially  important  to  remove  unmotivated  subjects  from  validation  studies  involving 
tests  of  processing  speed 
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